Due: 8:00pm 29 May 2022 (Monday)
This is an individual assignment. It contributes 40% to your final mark. Read the assignment instruction carefully.This assignment is to be completed individually and submitted to CloudDeakin. By the due date, you are required to submit the following files to the corresponding Assignment (Dropbox) in CloudDeakin:
For example, if your student ID is: 123456, you will then need to submit the following files:
Your submission will be assessed based on the overall impact of your effort. A useful model (or application) should be your focus. But as in Assignment 1, we will also consider the following criteria at the same time.
This assignment is to feedback on your learning in deep learning theory and its application to data analytics or artificial intelligence problems.
It builds on Assignment 1 but requires a higher level of mastery of deep learning theory and programming/engineering skills. In particular, you will practice making design decisions yourself. You are also likely to encounter practical issues that will help you consolidate textbook learning.
In Assignment 1, you tackled the image classification problem in Fashion-MNIST. There, you used a Densely Connected Neural Network. In Assignment 2, you will apply the best practices of deep-learning computer vision to make something useful for our planet—waste classification.
Background Every day, we put things into our recycle bin, to reduce landfill waste. However, we may unintentionally contribute to recycling contamination by "wish recycling" the wrong items. As every city council has slightly different rules for recycling, you will build a technological solution to ensure you only recycle things that are permitted by your local council. More discussions about recycling contamination can be found here.

Define an image classification problem that may help people better recycle, particularly by reducing contamination.
Describe the desired inputs and outputs, including the target classes.
What dataset can you use to develop a deep learning solution?
How many images do you need? How many for training? How many for testing?
Do you need to label the images yourself?
How do you determine if your model is good enough?
Collect relevant data. Develop a deep learning model. Report the model performance against the success criteria that you define.
Importing Necessary Packages
import pandas as pd
import numpy as np
import os
import glob
from matplotlib.image import imread
from matplotlib import pyplot as plt
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import tensorflow as tf
import datetime as dt
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
A function to get the train and test dataset along with distribution of data
def get_data(dir, t_size, b_size, kwargs_idg = {}, kwargs_ffd = {}):
data_gen = ImageDataGenerator(rescale=1./255, **kwargs_idg)
data = data_gen.flow_from_directory(os.path.join("archive/DATASET/","DATASET",dir), target_size=t_size, batch_size=b_size, **kwargs_ffd)
return data
train = get_data("TRAIN", (64,64), 128, {"validation_split" : 0.1}, {"subset":"training"})
validation = get_data("TRAIN", (64,64), 128, {"validation_split" : 0.1}, {"subset":"validation"})
test = get_data("TEST", (64,64), 128)
Found 20309 images belonging to 2 classes. Found 2255 images belonging to 2 classes. Found 2513 images belonging to 2 classes.
Looking into number of classes in the data
train.class_indices
{'O': 0, 'R': 1}
Showing a total of fourty images from the training set
img_data, _= next(train)
plt.figure(figsize=(25, 16))
for j in range(40):
ax = plt.subplot(8, 8, j + 1)
plt.imshow(img_data[j])
if _[j][1] == 0:
plt.title("Organic")
else:
plt.title("Recyclable")
plt.axis("off")
del img_data,_
Defining a model for the dataset
def model(input_size, data_aug = 0):
model = tf.keras.Sequential()
if data_aug == 1:
data_augmentation = tf.keras.Sequential(
[
tf.keras.layers.experimental.preprocessing.RandomFlip("horizontal",
input_shape=input_size),
tf.keras.layers.experimental.preprocessing.RandomRotation(0.1),
tf.keras.layers.experimental.preprocessing.RandomZoom(0.1),
tf.keras.layers.experimental.preprocessing.RandomContrast(factor=(0.8, 1.4))
]
)
model.add(data_augmentation)
# Add convolutional layers
model.add(tf.keras.layers.Conv2D(8, (3, 3), activation='relu', input_shape=input_size))
model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(256, input_shape=input_size, activation="sigmoid"))
model.add(tf.keras.layers.Dense(256, activation='relu', kernel_regularizer=tf.keras.regularizers.L1L2(l2=0.001)) )
model.add(tf.keras.layers.Dense(128, activation='relu', kernel_regularizer=tf.keras.regularizers.L1L2(l2=0.001)))
model.add(tf.keras.layers.Dense(2, activation='softmax'))
model.compile(optimizer='Nadam',loss='CategoricalCrossentropy', metrics=['accuracy'])
return model
Defining a tensorboard callback to look into the tensorboard profile and model analysis
# Create a TensorBoard callback
logs = "logs/" + dt.datetime.now().strftime("%Y%m%d-%H%M%S")
tboard_callback = tf.keras.callbacks.TensorBoard(log_dir = logs,
histogram_freq = 1,
profile_batch = '500,520')
Training the model
model_nn = model((64, 64, 3))
history = model_nn.fit(train, validation_data=validation, epochs=15, callbacks=[tboard_callback])
Epoch 1/15 159/159 [==============================] - 350s 2s/step - loss: 0.7437 - accuracy: 0.7533 - val_loss: 0.5052 - val_accuracy: 0.8333 Epoch 2/15 159/159 [==============================] - 248s 2s/step - loss: 0.5515 - accuracy: 0.8040 - val_loss: 0.4663 - val_accuracy: 0.8368 Epoch 3/15 159/159 [==============================] - 61s 384ms/step - loss: 0.4881 - accuracy: 0.8168 - val_loss: 0.4046 - val_accuracy: 0.8541 Epoch 4/15 159/159 [==============================] - 44s 274ms/step - loss: 0.4384 - accuracy: 0.8329 - val_loss: 0.4241 - val_accuracy: 0.8501 Epoch 5/15 159/159 [==============================] - 127s 799ms/step - loss: 0.3966 - accuracy: 0.8506 - val_loss: 0.3777 - val_accuracy: 0.8594 Epoch 6/15 159/159 [==============================] - 65s 411ms/step - loss: 0.3645 - accuracy: 0.8608 - val_loss: 0.3623 - val_accuracy: 0.8674 Epoch 7/15 159/159 [==============================] - 41s 260ms/step - loss: 0.3312 - accuracy: 0.8770 - val_loss: 0.3786 - val_accuracy: 0.8585 Epoch 8/15 159/159 [==============================] - 50s 313ms/step - loss: 0.2986 - accuracy: 0.8901 - val_loss: 0.3692 - val_accuracy: 0.8608 Epoch 9/15 159/159 [==============================] - 36s 226ms/step - loss: 0.2674 - accuracy: 0.9043 - val_loss: 0.3952 - val_accuracy: 0.8665 Epoch 10/15 159/159 [==============================] - 66s 418ms/step - loss: 0.2324 - accuracy: 0.9198 - val_loss: 0.4822 - val_accuracy: 0.8373 Epoch 11/15 159/159 [==============================] - 34s 217ms/step - loss: 0.1948 - accuracy: 0.9360 - val_loss: 0.5174 - val_accuracy: 0.8324 Epoch 12/15 159/159 [==============================] - 30s 190ms/step - loss: 0.1563 - accuracy: 0.9521 - val_loss: 0.5169 - val_accuracy: 0.8355 Epoch 13/15 159/159 [==============================] - 32s 202ms/step - loss: 0.1167 - accuracy: 0.9702 - val_loss: 0.5774 - val_accuracy: 0.8408 Epoch 14/15 159/159 [==============================] - 31s 198ms/step - loss: 0.1100 - accuracy: 0.9715 - val_loss: 0.6487 - val_accuracy: 0.8421 Epoch 15/15 159/159 [==============================] - 30s 186ms/step - loss: 0.0954 - accuracy: 0.9782 - val_loss: 0.5058 - val_accuracy: 0.8293
Model Summary
model_nn.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 62, 62, 8) 224
max_pooling2d (MaxPooling2D (None, 31, 31, 8) 0
)
flatten (Flatten) (None, 7688) 0
dense (Dense) (None, 256) 1968384
dense_1 (Dense) (None, 256) 65792
dense_2 (Dense) (None, 128) 32896
dense_3 (Dense) (None, 2) 258
=================================================================
Total params: 2,067,554
Trainable params: 2,067,554
Non-trainable params: 0
_________________________________________________________________
Model Accuracy Visualization
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()
Build a data preprocessing pipeline to perform data augmentation. (You may use Keras ImageDataGenerator or write your own transformations.)
Report the model performance with the pipeline added. How much performance gain have you achieved?
Profile your input pipeline to identify the most time-consuming operation. What actions have you taken to address that slow operation? (Hint: You may use a profiler such as the TensorFlow Profiler.)
Training the model using data augumentation
new_model = model((64,64,3), 1)
history_new = new_model.fit(train, validation_data=validation, epochs=15, callbacks=[tboard_callback])
Epoch 1/15 159/159 [==============================] - 157s 956ms/step - loss: 0.7359 - accuracy: 0.7603 - val_loss: 0.5332 - val_accuracy: 0.8377 Epoch 2/15 159/159 [==============================] - 100s 630ms/step - loss: 0.5693 - accuracy: 0.7929 - val_loss: 0.4610 - val_accuracy: 0.8435 Epoch 3/15 159/159 [==============================] - 40s 248ms/step - loss: 0.5162 - accuracy: 0.7968 - val_loss: 0.4294 - val_accuracy: 0.8333 Epoch 4/15 159/159 [==============================] - 45s 281ms/step - loss: 0.4894 - accuracy: 0.7977 - val_loss: 0.4030 - val_accuracy: 0.8435 Epoch 5/15 159/159 [==============================] - 42s 264ms/step - loss: 0.4691 - accuracy: 0.8022 - val_loss: 0.3914 - val_accuracy: 0.8479 Epoch 6/15 159/159 [==============================] - 36s 224ms/step - loss: 0.4531 - accuracy: 0.8054 - val_loss: 0.3967 - val_accuracy: 0.8439 Epoch 7/15 159/159 [==============================] - 32s 204ms/step - loss: 0.4466 - accuracy: 0.8080 - val_loss: 0.3960 - val_accuracy: 0.8452 Epoch 8/15 159/159 [==============================] - 37s 234ms/step - loss: 0.4384 - accuracy: 0.8081 - val_loss: 0.3750 - val_accuracy: 0.8559 Epoch 9/15 159/159 [==============================] - 34s 215ms/step - loss: 0.4280 - accuracy: 0.8139 - val_loss: 0.3797 - val_accuracy: 0.8510 Epoch 10/15 159/159 [==============================] - 38s 238ms/step - loss: 0.4221 - accuracy: 0.8149 - val_loss: 0.3840 - val_accuracy: 0.8483 Epoch 11/15 159/159 [==============================] - 38s 242ms/step - loss: 0.4169 - accuracy: 0.8163 - val_loss: 0.3762 - val_accuracy: 0.8514 Epoch 12/15 159/159 [==============================] - 33s 205ms/step - loss: 0.4130 - accuracy: 0.8180 - val_loss: 0.3836 - val_accuracy: 0.8479 Epoch 13/15 159/159 [==============================] - 33s 205ms/step - loss: 0.4063 - accuracy: 0.8225 - val_loss: 0.3810 - val_accuracy: 0.8488 Epoch 14/15 159/159 [==============================] - 32s 203ms/step - loss: 0.4023 - accuracy: 0.8238 - val_loss: 0.3639 - val_accuracy: 0.8483 Epoch 15/15 159/159 [==============================] - 33s 209ms/step - loss: 0.3976 - accuracy: 0.8260 - val_loss: 0.3610 - val_accuracy: 0.8559
Model Summary
new_model.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
sequential_2 (Sequential) (None, 64, 64, 3) 0
conv2d_1 (Conv2D) (None, 62, 62, 8) 224
max_pooling2d_1 (MaxPooling (None, 31, 31, 8) 0
2D)
flatten_1 (Flatten) (None, 7688) 0
dense_4 (Dense) (None, 256) 1968384
dense_5 (Dense) (None, 256) 65792
dense_6 (Dense) (None, 128) 32896
dense_7 (Dense) (None, 2) 258
=================================================================
Total params: 2,067,554
Trainable params: 2,067,554
Non-trainable params: 0
_________________________________________________________________
Visualizing the Training and Validation Accuracy
plt.plot(history_new.history['accuracy'])
plt.plot(history_new.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()
# Load the TensorBoard notebook extension.
%load_ext tensorboard
# Launch TensorBoard and navigate to the Profile tab to view performance profile
%tensorboard --logdir=logs
You may notice that with your pipeline, the model performance improves, but at the cost of a longer training time per epoch. Is the additional training time well spent? Compare the dynamic of model performance (e.g., classification accuracy on the test data) with and without data augmentation, when equal training time is spent in the two scenarios.
Model evaluation without Data Augumentation
score = model_nn.evaluate(test, verbose=0)
print('Test Loss', score[0])
print('Test accuracy', score[1])
Test Loss 0.5492393374443054 Test accuracy 0.8320732116699219
Model evaluation with Data Augumentation
score_da = new_model.evaluate(test, verbose=0)
print('Test with DA Loss', score_da[0])
print('Test with DA accuracy', score_da[1])
Test with DA Loss 0.37655389308929443 Test with DA accuracy 0.844011127948761
Identify images that are incorrectly classified by your model. Do they share something in common? How do you plan to improve the model's performance on those images?
Actual vs Predicted wiothout DA
test_x, test_y = test.__getitem__(1)
labels = (test.class_indices)
labels = dict((v,k) for k,v in labels.items())
preds = model_nn.predict(test_x)
plt.figure(figsize=(16, 16))
j=0
for i in range(64):
if labels[np.argmax(preds[i])]!=labels[np.argmax(test_y[i])]:
plt.subplot(4, 4, j+1)
plt.title('pred:%s / actual:%s' % (labels[np.argmax(preds[i])], labels[np.argmax(test_y[i])]))
plt.imshow(test_x[i])
j = j+1
4/4 [==============================] - 0s 8ms/step
Actual vs Predicted With DA
test_x, test_y = test.__getitem__(1)
labels = (test.class_indices)
labels = dict((v,k) for k,v in labels.items())
preds = new_model.predict(test_x)
plt.figure(figsize=(16, 16))
j=0
for i in range(64):
if labels[np.argmax(preds[i])]!=labels[np.argmax(test_y[i])]:
plt.subplot(4, 4, j+1)
plt.title('pred:%s / actual:%s' % (labels[np.argmax(preds[i])], labels[np.argmax(test_y[i])]))
plt.imshow(test_x[i])
j = j+1
4/4 [==============================] - 0s 8ms/step
So far, you have used training and test images from the same source (via random data split). Now collect new test images from a different source. For example, you may take some photos yourself if you used downloaded images before. Otherwise, you may take new photos using a different mobile phone or against a different background.
Show sample images from the original test data and the newly collected test data. In what ways are they different?
Feed the new test data into your model. Report the performance change.
Improve your model so that it generalises better on unseen test images.
Generating the data
test_new = get_data("NEWTEST", (64,64), 128)
Found 1523 images belonging to 2 classes.
Visualizing a sample of test dataset
test_new_x, test_new_y = test_new.__getitem__(1)
labels_new = (test_new.class_indices)
labels_new = dict((v,k) for k,v in labels_new.items())
plt.figure(figsize=(16, 16))
for i in range(16):
plt.subplot(4, 4, i+1)
plt.title('Label:%s' % (labels[np.argmax(test_new_y[i])]))
plt.imshow(test_new_x[i])
Evaluating model without DA performance
score_new = model_nn.evaluate(test_new, verbose=0)
print('New Test', score_new[0])
print('New Test', score_new[1])
New Test 0.746645450592041 New Test 0.7071569561958313
Evaluating model with DA performance
score_new_da = new_model.evaluate(test_new, verbose=0)
print('New Test with DA Loss', score_new_da[0])
print('New Test with DA accuracy', score_new_da[1])
New Test with DA Loss 0.7233662009239197 New Test with DA accuracy 0.6178594827651978
Actual vs Predicted without DA
preds = model_nn.predict(test_new_x)
plt.figure(figsize=(16, 16))
for i in range(16):
plt.subplot(4, 4, i+1)
plt.title('pred:%s / actual:%s' % (labels[np.argmax(preds[i])], labels[np.argmax(test_new_y[i])]))
plt.imshow(test_new_x[i])
4/4 [==============================] - 0s 8ms/step
Actual vs Predicted with DA
preds = new_model.predict(test_new_x)
plt.figure(figsize=(16, 16))
for i in range(16):
plt.subplot(4, 4, i+1)
plt.title('pred:%s / actual:%s' % (labels[np.argmax(preds[i])], labels[np.argmax(test_new_y[i])]))
plt.imshow(test_new_x[i])
4/4 [==============================] - 0s 8ms/step
Applying Batch Normalization to our model to imporve the model performence
def model_improve(input_size, data_aug = 0):
model = tf.keras.Sequential()
if data_aug == 1:
data_augmentation = tf.keras.Sequential(
[
tf.keras.layers.experimental.preprocessing.RandomFlip("horizontal",
input_shape=input_size),
tf.keras.layers.experimental.preprocessing.RandomRotation(0.1),
tf.keras.layers.experimental.preprocessing.RandomZoom(0.1),
tf.keras.layers.experimental.preprocessing.RandomContrast(factor=(0.8, 1.4))
]
)
model.add(data_augmentation)
# Add convolutional layers
model.add(tf.keras.layers.Conv2D(8, (3, 3), activation='relu', input_shape=input_size))
model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Dense(256, input_shape=input_size, activation="sigmoid")) # Input Layer
model.add(tf.keras.layers.Dense(256, activation='relu', kernel_regularizer=tf.keras.regularizers.L1L2(l2=0.001)) )
model.add(tf.keras.layers.Dense(128, activation='relu', kernel_regularizer=tf.keras.regularizers.L1L2(l2=0.001)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Dense(2, activation='softmax'))
model.compile(optimizer='Nadam',loss='CategoricalCrossentropy', metrics=['accuracy'])
# print(model.summary())
return model
imp_model = model_improve((64,64,3))
history_new_imp = imp_model.fit(train, validation_data=validation, epochs=15, callbacks=[tboard_callback])
Epoch 1/15 159/159 [==============================] - 223s 1s/step - loss: 0.7621 - accuracy: 0.8115 - val_loss: 0.7315 - val_accuracy: 0.8053 Epoch 2/15 159/159 [==============================] - 63s 395ms/step - loss: 0.5635 - accuracy: 0.8575 - val_loss: 0.6765 - val_accuracy: 0.7778 Epoch 3/15 159/159 [==============================] - 39s 243ms/step - loss: 0.4324 - accuracy: 0.8920 - val_loss: 0.5489 - val_accuracy: 0.8302 Epoch 4/15 159/159 [==============================] - 44s 276ms/step - loss: 0.3198 - accuracy: 0.9236 - val_loss: 0.5993 - val_accuracy: 0.8186 Epoch 5/15 159/159 [==============================] - 44s 274ms/step - loss: 0.2348 - accuracy: 0.9491 - val_loss: 0.7486 - val_accuracy: 0.7734 Epoch 6/15 159/159 [==============================] - 42s 262ms/step - loss: 0.1805 - accuracy: 0.9633 - val_loss: 0.7226 - val_accuracy: 0.8089 Epoch 7/15 159/159 [==============================] - 122s 772ms/step - loss: 0.1483 - accuracy: 0.9698 - val_loss: 0.7027 - val_accuracy: 0.8071 Epoch 8/15 159/159 [==============================] - 268s 2s/step - loss: 0.1206 - accuracy: 0.9766 - val_loss: 0.8517 - val_accuracy: 0.7818 Epoch 9/15 159/159 [==============================] - 305s 2s/step - loss: 0.1037 - accuracy: 0.9794 - val_loss: 0.7777 - val_accuracy: 0.8040 Epoch 10/15 159/159 [==============================] - 70s 439ms/step - loss: 0.0940 - accuracy: 0.9815 - val_loss: 0.8899 - val_accuracy: 0.8058 Epoch 11/15 159/159 [==============================] - 33s 209ms/step - loss: 0.0892 - accuracy: 0.9799 - val_loss: 0.8576 - val_accuracy: 0.7947 Epoch 12/15 159/159 [==============================] - 34s 212ms/step - loss: 0.0799 - accuracy: 0.9824 - val_loss: 1.0717 - val_accuracy: 0.7969 Epoch 13/15 159/159 [==============================] - 34s 211ms/step - loss: 0.0667 - accuracy: 0.9866 - val_loss: 1.1512 - val_accuracy: 0.7809 Epoch 14/15 159/159 [==============================] - 34s 212ms/step - loss: 0.0632 - accuracy: 0.9871 - val_loss: 0.9133 - val_accuracy: 0.8102 Epoch 15/15 159/159 [==============================] - 36s 227ms/step - loss: 0.0630 - accuracy: 0.9865 - val_loss: 1.1094 - val_accuracy: 0.7734
Model accuracy on the test set of the original dataset
score_new = imp_model.evaluate(test, verbose=0)
print('New Test', score_new[0])
print('New Test', score_new[1])
New Test 0.7943583726882935 New Test 0.8312773704528809
Model accuracy on new data set
score_new = imp_model.evaluate(test_new, verbose=0)
print('New Test', score_new[0])
print('New Test', score_new[1])
New Test 0.8945891857147217 New Test 0.8154957294464111
Saving the current model as the best model
tf.keras.models.save_model(imp_model,'best_model.h5')
Build a web/mobile app that people from your city council can use to determine what to recycle. Test your prototype with the target users and report their feedback.
Upload your code into a GitHub repository.
Create a short video presentation about your product.
END OF ASSIGNMENT TWO